ar X iv : c on d - m at / 0 51 20 17 v 2 9 J an 2 00 6 Combinatorial Information Theory : I . Philosophical Basis of Cross - Entropy and Entropy

The three main theoretical bases of the concepts of entropy and cross-entropy informationtheoretic, axiomatic and combinatorial are critically examined. It is shown that the combinatorial basis, proposed by Boltzmann and Planck, is the most fundamental (most primitive) basis of these concepts, since it provides (i) a derivation of the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) the means to derive entropy and cross-entropy functions for systems which do not satisfy the multinomial distribution, i.e. which fall outside the domain of the Kullback-Leibler and Shannon measures. The information-theoretic and axiomatic bases of cross-entropy and entropy whilst of tremendous importance and utility are therefore seen as secondary viewpoints, which lack the breadth of the combinatorial approach. Appreciation of this reasoning would permit development of a powerful body of “combinatorial information theory”, as a tool for statistical inference in all fields (inside and outside science). The essential features of Jaynes’ analysis of entropy and cross-entropy reinterpreted in light of the combinatorial approach are outlined, including derivation of probability distributions, ensemble theory, Jaynes relations, fluctuation theory and Jaynes’ entropy concentration theorem. New results include a generalized free energy (or “free information”) concept, a generalized Gibbs-Duhem relation and phase rule. Generalized (combinatorial) definitions of entropy and cross-entropy, valid for any combinatorial system, are then proposed and examined in detail. PACS numbers: 02.50.Cw, 02.50.Tt, 05.20.-y, 05.20.Gg, 05.30.-d, 05.30.Ch, 05.40.-a, 05.70.-a, 05.70.Ce, 05.90.+m, 64.10.+h, 89.20.-a, 89.70.+c,